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Abstract 

Background and objective: The aim of this study was to develop a questionnaire for assessing nutritional 
knowledge among overweight adults. The questionnaire should reveal knowledge about current dietary 
recommendations, sources of nutrients, everyday food choices, and conditions related to overweight. 
Design: The first draft of the nutrition knowledge questionnaire (113 items) was based on literature review. 
To ensure content validity and expert-assessed face validity, an expert panel examined the questionnaire. 
Thereafter, the questionnaire was tested for user friendliness and ambiguity by five students. The 
questionnaire was pilot tested in a group of obese adults, similar to the target group. The results were 
analyzed for item difficulty and internal consistency and comments made by respondents were taken into 
account. Two student groups, differing in nutritional expertise, answered the questionnaire on two occasions 
to test construct validity and test-retest reliability. After the retest, a total overview of the questionnaire 
was made by the expert panel. The final questionnaire consisted of 91 items. 

Subjects: The pilot study was conducted in obese adults waiting for a gastric bypass operation («=33). 
Construct validity (n =34) and test-retest reliability (n =27) was tested in two student groups: public health 
nutrition students and construction students. 

Results: Results from the pilot study showed that internal consistency of the three first sections together 
was 0.84, measured by Cronbach's ot. Test of construct validity showed that public health nutrition students 
scored significantly better than construction students (/><0.001 for all sections), and test-retest reliability 
for all sections together was 0.82 (Pearson's r). 

Conclusion: The knowledge questionnaire had reasonable content-, face-, and construct validities and overall 
good reliability. The questionnaire can be a useful tool for measuring nutrition knowledge among obese 
adults. 
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Obesity is a global challenge and one of the 
main risk factors for disease and early death 
(1). The condition increases the risk for develop- 
ing diabetes type 2, cardiovascular disease, high-blood 
pressure, and several cancers (1-4). Rates of obesity are 
on the rise in both developed and developing countries 
(5). This is caused by an imbalance between energy intake 
and expenditure, due to increased intakes of energy- 
dense foods and reduced levels of physical activity in 
parts of the population (1, 4). Modifications of these 
behaviours are therefore needed in order to halt the 
increase in obesity. One of the suggested strategies for 
behavior modification is to enable people to make 
informed choices and take effective action (6). 



Knowledge is recognized to be one of the components 
for change of food habits (7, 8) but research is, however, 
inconclusive on how important nutrition knowledge 
is for changing dietary behavior (8-14). One of the 
explanations for the inconclusive findings could be 
that knowledge has been poorly measured (15, 16). 

Knowledge can be difficult to assess and good instru- 
ments for valid and reliable measurements are required 
(17, 18). Kline had defined a set of criteria for construct- 
ing a valid and reliable test that can be used for 
measuring psychological attributes, including nutrition 
knowledge (19). Parmenter and Wardle (20) focused on 
psychometric measures when developing their question- 
naire to measure general nutrition knowledge among 
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adults. Furthermore, they have developed a guide for 
evaluation and design of nutrition knowledge measures 
(15). Psychometric methods have been used by seve- 
ral in development of existing nutrition knowledge 
questionnaires (9, 11, 18, 20-23). 

The aim of the present study was to develop a nutrition 
knowledge questionnaire with satisfactory reliability and 
validity for use among obese adults. The questionnaire 
should assess the level of general nutrition knowledge 
among obese adults about current dietary recommenda- 
tions, sources of nutrients, everyday food choices, and 
assess the knowledge of different topics related to 
obesity. This instrument can be used in a survey setting 
but also as a starting point for tailoring educational 
programs in smaller groups and clinical practice. Since 
the nutrition knowledge can be variable among patients 
and group participants, mapping of preknowledge can 
make the teaching more relevant, focused, and effective. 

Methods and results 

The questionnaire development process consisted of four 
steps (Fig. 1): (1) preparation of scope and structure; (2) 
development of questionnaire items; (3) pilot study for 
further development of the questionnaire; and (4) test 
and retest by student groups for construct validity 
assessment and reproducibility. 

The development and evaluation of the questionnaire 
was conducted in 2006-2007. Norwegian Social Science 
Data Services approved the study and The Regional 
Committees for Medical and Health Research Ethics 
approved the pilot study. Informed consent was obtained 
from all participants. 

Step I - Preparation of scope and structure 

A literature review of nutrition knowledge linked to 
weight reduction, dietary behavior, general dietary re- 
commendations, and nutrition knowledge questionnaires 
was performed to define the scope of the questionnaire. 
In addition, telephone calls and personal meetings 
were held with professionals working in the area of 
dietetics and obesity to get in-depth information of 
the different aspects the questionnaire should cover. 
Based on this information, it was decided to divide 
the questionnaire into four main sections assessing 
knowledge about: (1) official dietary recommendations; 
(2) nutrient content in food items, with a main focus on 
macronutrients; (3) healthy food choices; and (4) meal 
pattern, energy intake, food labeling, and the food plate 
model. The first three sections were related to items about 
general nutrition knowledge. The fourth section was 
specifically related to obesity (such as calorific require- 
ments, sugar contents of soft drinks, and skills of label 
reading). As a fifth section, demographic questions were 
included at the end of the questionnaire (age, gender, 
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educational level, nutrition-related qualifications, occu- 
pation, height, and weight). 

Step 2 - Development of questionnaire items 

Based on the literature review and the four chosen 
categories, an item pool of 273 items was generated, 
mainly based on six existing nutrition knowledge ques- 
tionnaires (9, 11, 16, 20, 23, and AKH de Soysa & 
S Severinsson, personal communication, 2006). 

Content validity refers to how representative the items 
are in covering the subject matter. The items should, 
as far as possible, sample the whole domain of the 
construct (16, 22). The items in the present knowledge 
questionnaire should cover essential aspects of general 
nutrition knowledge and conditions related to obesity 
and be phrased simply and unambiguously (20). 

Face validity refers to how relevant and appropriate 
items appear to the respondents (15). Persons who have 
expertise with the target group or representatives of 
the target group itself are generally considered to be 
good judges of face validity (21, 24). 

To ensure content validity and face validity, the item 
pool was evaluated by an expert panel of four dieticians 
who had experience with weight reduction groups. The 
experts received the item pool of 273 items and were 
requested to evaluate them for accuracy, appropriateness, 
and relevance related to measuring nutrition knowledge 
among obese. The expert panel selected 113 items from 
the item pool, and these items became the first ques- 
tionnaire draft. The knowledge items were divided into 
three main types of answering options: agree/disagree 
(/not sure), multiple-choice, and more/less of different 
food items. These 113 items were then subjected to a 
second evaluation by the expert panel. The experts were 
in the second round asked to select items for adequate 
coverage of the knowledge area, interpretability of the 
food choices listed, and structure of the different ques- 
tionnaire sections. Other complementary items were 
also asked for. In addition, one expert of questionnaire 
development was consulted to improve structure and 
layout. 

After this review of the first draft, the answer 
category 'not sure' was included for items in sections 
for dietary recommendations and sources of nutrients. 
The expert panel review led to changes in 23 items, and 
the second draft of the questionnaire consisted of 98 
items. Change of items included editing of items, remov- 
ing, and adding new items. The second draft was 
tested for user friendliness and item ambiguity in a 
prepilot conducted by five students who had completed 
a course in general nutrition. Each questionnaire item 
was orally asked and the students discussed the answers. 
The discussion was taped. The prepilot led to changes 
in five items. After a new consultation with the experts, 
a third draft was finalized with 98 items. 
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Step 4. Test and retest by student groups for construct validity assessment and reproducibility 
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Final questionnaire: 91 items with content validity, face validity, construct validity, as well as internal 
consistency and test-retest reliability 



Fig. J. Summary of steps followed in the development of the nutrition knowledge questionnaire for obese adults (figure inspired 
by Whati et al., 22). 
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Step 3 - Pilot study in target group for further development 
of the questionnaire 

A pilot study was conducted to test whether the ques- 
tionnaire was appropriate in a group of obese adults 
(«=33, BMI>35, age between 21 and 60 years). 
The respondents, similar to the target group, participated 
in a nutrition course prior to gastric bypass operation. 
Data were analyzed using SPSS version 14.0 for Windows 
(SPSS Inc., Chicago, IL, USA). The results were analyzed 
quantitatively for item difficulty and internal consistency 
and qualitatively by looking at the respondents' com- 
ments on, e.g. the format of the questionnaire, the 
interpretability of the item, lack of important items, and 
time used for filling in the questionnaire. 

Item difficulty refers to the proportion of respondents 
answering items correctly. Individual items should not 
be so easy that almost everyone completes them, nor 
so difficult that very few complete them (19, 20). 
The item analyses from the pilot study showed for 
example that all (100%) agreed that they should eat 
more vegetables and less sugar (94%). Few had the 
knowledge that olive oil (39%) and vegetable margarine 
(36%) contained a lot of fat. This was in accordance 
with the result that only 9% knew that it was the same 
amount of calories in butter and margarine. Only 36% 
disagreed on the statement that 'oil contains less fat 
than vegetable margarine.' Few (27%) also disagreed that 
dark chocolate contained less fat than milk chocolate. 
These results were concurrent with the comments from 
the participants in the prepilot test. 

Kline (19) claimed that items are not useful if they 
are answered correctly by fewer than 20% or more than 
80% of respondents (20). The limits of 30%-90% was 
chosen for this study to let less knowledgeable respon- 
dents be able to answer more items as a motivation 
factor for completing the questionnaire. Based on the 
pilot study, 22 items were answered correctly by more 
than 90% and five items by fewer than 30% of the 
respondents. Of the 27 items that did not meet the item 
difficulty criteria, 19 items were nevertheless retained to 
assess essential aspects of nutrition knowledge not 
covered elsewhere in the questionnaire. 

Internal consistency refers to the homogeneity of the 
questionnaire. Individual items within each section or 
subsections should be well correlated to the total 
score of all sections (20, 25). Internal consistency was 
measured using Cronbach's a. Cronbach's a values 
range from 0 to 1, and a score of 0.7 or higher is 
generally acceptable (20, 25). All variables were recoded 
into 1 or 0, where 1 = correct, 0 = wrong, or 0 =not sure. 
Cronbach's a was calculated for each of the three first 
sections (dietary recommendations, sources of nutrients, 
and healthy food choices) and for the three sections 
combined (Table 1). Cronbach's a varied from 0.51 in 
the section for healthy food choices to 0.84 for the 
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Table 1. Internal consistency in pilot study of obese adults (n = 33) 





Internal consistency 


Knowledge section (number of items) 


(Cronbach's a) 


1. Dietary recommendations (II) 


0.66 


2. Sources of nutrients (55) 


0.80 


3. Healthy food choices (21) 


0.51 


Total (section 1-3) (87) 


0.84 



three first sections combined (Table 1). The fourth 
section (factors that relate to obesity reduction) consisted 
of different types of items, as meal frequency, counting 
of calories, food labeling, and one would not assume 
any internal consistency between these items. 

A further analysis of items in sections for dietary 
recommendations and healthy food choices was con- 
ducted because these sections did not reach the score 
of 0.7. This analysis showed for example that the 
items 'How many servings of fruit and vegetables 
should we eat every day?' (correct answer 67%) and 
'What would be best to drink when you are thirsty? 
Skimmed milk, orange juice, water or low calorie soft 
drink' (correct answer 100%), would give their sections 
a higher Cronbach's a if deleted. Totally, there were 
13 items that produced a higher a value for the total 
section when deleted; however, only three were removed. 
The rest was retained for the sake of content validity (22). 

Respondents' comments were made from 25 of the 33 
respondents. Time used for filling in the questionnaire 
was between 10 and 45 min. Several of the participants 
commented that items regarding carbohydrates and 
proteins were difficult to answer. One remarked that 
'it is a bit tricky when the food options seem to be 
exactly the same.' However, 10 respondents answered 
'no' to the question 'did you think that some of the 
items were difficult?' The respondents were also asked 
to comment on positive aspects of the questionnaire. 
Representative remarks were T think that the most 
important issues are covered' and 'we get an eye-opener 
about how little we care about diet, which is so 
important.' 

The results from the measures of item difficulty 
and internal consistency from the pilot study and 
comments from the evaluation of the questionnaire 
were all considered before further changes were done. 
After consultations with the expert panel, the fourth 
draft of the questionnaire was finalized. Between the 
third and fourth draft, 18 items were changed and 
the fourth draft consisted of 92 items. 

Step 4 - Test and retest study 

A test-retest study was conducted with two groups of 
students; one group consisted of second year bachelor 
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students in public health nutrition (»=16) and the 
other group consisted of construction students (« = 18). 
The number of students who completed both test and 
retest was 27. 

Construct validity refers to whether the combination 
of items in a specific construct provides a good measure 
(15, 22). The construct validity can be assessed in two 
groups, one which is known to have good nutrition 
knowledge, e.g. public health nutrition students and the 
other not e.g. construction students. The former group 
should score significantly higher, which was ascertained 
statistically using an independent sample's f-test (11, 20, 
21). Table 2 shows the differences in knowledge between 
the public health nutrition and the construction students 
based on the first questionnaire test. The public health 
nutrition students had a significantly higher mean score 
in each section of the nutrition knowledge question- 
naire (/?<0.05) and for all sections together (p<0.001) 
(Table 2). 

The test— retest reproducibility of a questionnaire is 
the extent to which it produces the same results, when 
applied repeatedly in the same situation with the same 
persons (26). The time lag between the measures should 
be long enough for precise answers to be forgotten 
and short enough to minimize real change of knowledge 
(20). Pearson's correlation was used to measure the 
reliability between test and retest. It has been suggested 
that Pearson's correlation between scores of two tests 
should be at least 0.7 (15, 19). The time between the 
test and the retest was 3 and 4 weeks for the public 
health nutrition students and the construction students, 
respectively. Table 3 shows Pearson's correlation between 
the students who completed both the test and the 
retest. The correlation coefficients varied within sections 
from 0.53 (obesity reduction factors) to 0.85 (sources 
of nutrients). For all sections together, the correlation 
coefficient was 0.82. All correlations were significant for 
each section in the questionnaire (p <0.01) and for 
all sections together (p<0.001). After the retest, a 
total overview of the questionnaire was made by the 
expert panel. This resulted in three changes, and the 
final questionnaire consisted of 91 items (web link to 



questionnaire in Norwegian and a version translated 
to English: http://hist.no/content/33064/). 

Discussion 

In the present study, special attention was given to 
the development and testing of a nutrition knowledge 
questionnaire focusing on general dietary recommenda- 
tions (27) and food choices to be used among obese 
adults. 

Securing high-content validity was prioritized during 
the whole development process. The initial item pool 
consisted of items that already had been through a 
process considering reliability and validity (9, 11, 20, 
23). New items were added to the questionnaire if the 
experts considered that certain aspects were not covered. 
Every draft was reviewed by the expert panel to select the 
best items in terms of clarity of the items, interpretability, 
and accuracy of the knowledge area being measured. 

Separating the questionnaire into four main sections 
gives the possibility to assess both general and more 
specific nutrition knowledge related to obesity. The 
fourth section with specific items will also make the 
questionnaire more user friendly for other purposes. It is 
thereby easier for other users of the questionnaire to 
change the questions and direct the fourth section to for 
example diet-disease associations, fruit and vegetable 
intake, or diet recommendations for more specific groups. 

It has been stated that without face validity, it is 
questionable whether the measure has content validity 
(15). In the current study, considerable effort was invested 
to ensure face validity. This was done both by taping and 
analyzing the discussion of all items and answers in the 
prepilot study,asking the respondents in the pilot study to 
comment on the content and design of the questionnaire, 
and noting misunderstandings according to comprehen- 
sion (28). 

Content validity and face validity must be assessed 
qualitatively, whereas construct validity can be measured 
statistically. The public health nutrition students (with 
extensive nutritional knowledge) scored higher on all 
sections of the questionnaire compared to the construc- 
tion students. These significant differences signal that 



Table 2. Differences in knowledge scores between public health nutrition students and construction students in test (n = 34) 



Public health nutrition students (n = l6) Construction students (n = l8) 



Knowledge section (max score) Mean SD Mean SD 



1. Dietary recommendations (I I) 10. I* I.S 9.0 1.4 

2. Sources of nutrients (52) 43 .8*** 4.0 34.3 5. 1 

3. Healthy food choices (19) 16.1** 2.2 13.6 2.1 

4. Obesity reduction factors ( 1 0) 8.2*** I.I 6.9 I.I 
Total (92) 78.2*** 5.4 63.8 7.4 



Independent samples t-test, two-sided. *p<0.05, **p<0.0l, ***p<0.00l 
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Table 3. Test-retest reliability for public health nutrition students 
(n =13) and construction students (n = 14) 



Knowledge section (max score) 


Test-retest reliability 
(Pearson's r) 


1. Dietary recommendations (II) 


0.56** 


2. Sources of nutrients (52) 


0.85*** 


3. Healthy food choices (19) 


0.64*** 


4. Obesity reduction factors ( 1 0) 


0.53** 


Total (92) 


0.82*** 



**p<0.0l, ***p<0.00l 



the questionnaire measures what it is supposed to 
measure, namely different dimensions of nutrition knowl- 
edge (18, 20, 23). 

The sample size of both the pilot study («=33) 
and test-retest («=27) were relatively small. The relia- 
bility of the test depends on the sample size and test 
length (19). The reliability coefficient increases as the 
number of respondents and number of items increase. 
Separating the knowledge items of the questionnaire 
into smaller sections may have had an impact on the 
test results both for the internal consistency (Table 1), 
and the test-retest reliability of the questionnaire 
(Table 3). Measuring internal consistency for the pilot 
study, the sections for dietary recommendations (11 
items) and healthy food choices (21 items), did not 
meet the score of 0.7. The internal consistency measure 
was highest for the sources of nutrients section 
(55 items), and the overall Cronbach's a of 0.84 for 
these three sections together was high. Similar results are 
found by others (9, 20, 21, 25). The test-retest reliability 
of the questionnaire was significant within all sections. 
The test-retest reliability for the sections dietary recom- 
mendations (11 items), healthy food choices (19 items), 
and obesity reduction factors (10 items) did not reach 
the correlation score of 0.7 (Pearson's r). The correla- 
tion scores for all knowledge sections together (92 items) 
were 0.82 (Pearson's ;•), indicating that the questionnaire, 
seen as a whole, measures nutrition knowledge consis- 
tently over time from one testing occasion to another. 
This corresponds with what others have found for 
their knowledge questionnaires (18, 20, 25). 

Items, which are very intricate or very simple, may 
not differentiate between actual knowledge of indivi- 
duals (22). In this study, several items that did not have 
consistency with the rest of the questionnaire and items 
that did not meet difficulty criteria were retained for the 
sake of content validity and only few were removed. 
All these items were presented the expert panel before 
adjustments were made. One item the panel chose to 
remove was: 'What would be best to drink when you are 
thirsty?' because the item was not regarded as essential 



for mapping of knowledge. The expert panel chose on the 
other hand, to keep the items about eating more 
vegetables, eating less sugar and servings of fruit and 
vegetables because all the items were closely related to the 
dietary recommendations from the government. Another 
illustration is that less than 30% of the respondents 
answered correctly to the statement: 'Dark chocolate 
contain less fat than milk chocolate.' A reason for the 
misunderstanding could be the strong promotion of dark 
chocolate as a healthy product in the media. The item was 
on these grounds regarded as important for mapping this 
aspect of nutrition knowledge. 

This focus on content validity, by retaining items that 
did not meet the criteria for item difficulty and consis- 
tency, may influence the consistency of the questionnaire 
and hence the statistical result. However, the expert panel 
found it important to keep these items to be able to test 
essential aspects of nutrition (15, 16). 

Conclusions and implications 

This questionnaire was designed to assess nutrition 
knowledge among obese adults. The knowledge ques- 
tionnaire had reasonable content-, face-, and construct 
validities and overall good reliability. The questionnaire 
should provide a useful tool for measuring nutrition 
knowledge among obese adults. We believe that this tool 
also could be used among other groups of adults with 
minor adjustments. 
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